Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus includes a detection unit configured to detect two or more specific objects from a captured image, a first color estimation unit configured to estimate a first light source color from color information about a first specific object among the detected specific objects, a second color estimation unit configured to estimate a second light source color from color information about a second specific object among the detected specific objects, a third color estimation unit configured to estimate a third light source color from color information about an area including other than the detected specific objects, and a calculation unit configured to calculate a white balance correction value to be applied to the captured image by assigning a weight to at least one of the first, second, and third light source colors based on degrees of similarity between the first, second, and third light source colors.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing technique foradjusting white balance of an image.

Description of the Related Art

As an automatic white balance technique for an image processingapparatus, Japanese Patent Application Laid-Open No. 2010-187201discusses a technique for detecting a specific object by face detection,estimating a light source by using color information about the detectedarea, and calculating a white balance correction value. Deep learningtechniques have been paving ways to detection of various non-faceobjects such as skin and natural plants in recent years, and informationavailable for white balance calculation is ever increasing.

However, while deep learning techniques enable the detection of variousobjects, objects other than intended ones can be erroneously detected.An incorrect correction value is calculated if white balance iscalculated based on such an erroneous detection result.

SUMMARY

According to embodiments of the present disclosure, an image processingapparatus includes a detection unit configured to detect two or morespecific objects from a captured image, a first color estimation unitconfigured to estimate a first light source color from color informationabout a first specific object among the detected specific objects, asecond color estimation unit configured to estimate a second lightsource color from color information about a second specific object amongthe detected specific objects, a third color estimation unit configuredto estimate a third light source color from color information about anarea including other than the detected specific objects, and acalculation unit configured to calculate a white balance correctionvalue to be applied to the captured image by assigning a weight to atleast one of the first, second, and third light source colors based ondegrees of similarity between the first, second, and third light sourcecolors.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of an imagingapparatus including an image processing apparatus.

FIG. 2 is a diagram illustrating a configuration example of the imageprocessing apparatus.

FIG. 3 is a flowchart of processing to be performed by a centralprocessing unit (CPU) of the imaging apparatus.

FIG. 4 is a flowchart of white balance correction value calculationprocessing.

FIG. 5 is a diagram used to describe division of an image into blocks.

FIGS. 6A to 6C are graphs used to describe red/green (R/G) andblue/green (B/G) values and a black-body radiation locus.

FIGS. 7A to 7C are diagrams used to describe calculation of degrees ofsimilarity and degrees of reliability of light source colors.

FIG. 8 is a chart illustrating the degrees of similarity and the degreesof reliability of estimated light source colors in a table form.

FIG. 9 is a diagram used to describe recalculation of the degrees ofreliability and the estimated light source colors.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present disclosure will be described belowwith reference to the drawings. Note that the following exemplaryembodiments are not intended to limit the present disclosure, and allthe combinations of features described in some exemplary embodiments arenot necessarily indispensable to the present disclosure. Similarcomponents will be described with the same reference numerals.

FIG. 1 is a diagram illustrating a schematic configuration of an imagingapparatus 100 that is an application example of an image processingapparatus according to the present exemplary embodiment.

The imaging apparatus 100 is a camera such as a digital camera and adigital video camera. The imaging apparatus 100 may be an electronicdevice having a camera function, such as a mobile phone having a camerafunction or a computer with a built-in camera.

An optical system 101 includes a lens, a shutter, and a diaphragm. Theoptical system 101 forms an optical image of an object on an imagingplane of an image sensor 102. Information including a focal length, ashutter speed, and an aperture value is transmitted from the opticalsystem 101 to a central processing unit (CPU) 103. The image sensor 102is a charge-coupled device (CCD) image sensor or a complementarymetal-oxide-semiconductor (CMOS) image senor. The image sensor 102includes a red (R)-green (G)-blue (B) Bayer array, and converts theoptical image formed by the optical system 101 into pixel-by-pixelluminance information. Data digitized through a non-illustratedanalog-to-digital (AD) converter, i.e., raw data before developmentprocessing is stored into a primary storage device 104 via the CPU 103.An electrical gain (hereinafter, referred to as an InternationalOrganization for Standardization (ISO) speed) of the image sensor 102 isset by the CPU 103.

A photometric sensor 105 is divided into a plurality of photometricareas (for example, horizontally divided into 12 areas and vertically 8areas, a total of 96 areas), and detects object luminance in eachdivided area from the amount of light incident through the opticalsystem 101. Luminance signals of the respective photometric areas outputfrom the photometric sensor 105 are digitized by a non-illustrated ADconverter and transmitted to the CPU 103. The number of divided areasmay be any positive whole number and is not limited to 96 describedabove.

The CPU 103 functioning as a control unit implements functions of theimaging apparatus 100 by controlling the components of the imagingapparatus 100 based on input data and previously stored programs. In thefollowing description, at least some of the functions for the CPU 103 toimplement by executing the programs may be implemented by a dedicatedpiece of hardware such as an application specific integrated circuit(ASIC).

The primary storage device 104 is a volatile storage device such as arandom access memory (RAM), and used for operation of the CPU 103.Information stored in the primary storage device 104 can also be used byan image processing apparatus 106 or recorded on a recording medium 107.

A secondary storage device 108 is a nonvolatile storage device such asan electrically erasable programmable read-only memory (EEPROM). Thesecondary storage device 108 stores a program (firmware) and varioustypes of setting information for controlling the imaging apparatus 100.The program and various types of setting information stored in thesecondary storage device 108 is used by the CPU 103.

The recording medium 107 records data on images captured by the imagesensor 102 and temporarily stored in the primary storage device 104.Like a semiconductor memory card, the recording medium 107 is detachablefrom the imaging apparatus 100, and the data recorded on the recordingmedium 107 can be read by other devices such as a personal computer. Inother words, the imaging apparatus 100 includes a detachable mechanismand reading and writing functions for the recording medium 107.

A display unit 109 displays a viewfinder image during imaging, capturedimages, and a graphical user interface (GUI) image for interactiveoperations.

An operation unit 110 is an input device group for accepting a user'soperations and transmitting the operations to the CPU 103 as operationinput information. Examples of the input devices include a button, alever, and a touch panel. The operation unit 110 may include an inputdevice using voice or a line of sight. A release button from which theimaging apparatus 100 obtains a user operation to start imaging is alsoincluded in the operation unit 110. The imaging apparatus 100 accordingto the present exemplary embodiment has a plurality of patterns of imageprocessing for the image processing apparatus 106 to apply to a capturedimage. Such patterns can be set as imaging modes by using the operationunit 110.

The image processing apparatus 106 applies various types of imageprocessing to image data obtained by the imaging using the image sensor102. The image processing includes development processing such as whitebalance processing, color interpolation processing, gamma correctionprocessing, saturation correction processing, and hue correctionprocessing. The color interpolation processing is intended to convert anRGB Bayer array signal into R. G, and B, three color-specific planesignals. As will be described in detail below, the image processingapparatus 106 performs processing for detecting two or more specificobject areas from a captured image, specific object area colordetermination processing, light source color estimation processing,light source color similarity and reliability degree calculationprocessing, light source color mixing processing, and processing forcalculating white balance correction values. At least some of suchprocesses of the image processing apparatus 106 may be implemented bysoftware by the CPU 103 executing a program according to the presentexemplary embodiment.

Various types of calculation processing and control processing relatedto a white balance correction according to the present exemplaryembodiment will now be described.

FIG. 2 is a block diagram illustrating the functions of a white balancecontrol unit 200 that performs the white balance processing in the imageprocessing apparatus 106. In the following description, white balancewill be abbreviated as WB. The WB control unit 200 includes a divisionunit 201, a first color determination unit 202, a first color estimationunit 203, a second color determination unit 204, a second colorestimation unit 205, a white determination unit 206, a third colorestimation unit 207, a degree of similarity calculation unit 208, acolor mixing unit 209, and a correction value calculation unit 210.

The division unit 201 divides a captured image captured by the imagesensor 102 and temporarily stored in the primary storage device 104 intoa plurality of blocks. Details of the block division processing by thedivision unit 201 will be described below.

The first color determination unit 202 and the second colordetermination unit 204 detect respective specific objects from thecaptured image. In the present exemplary embodiment, for example, ahuman skin area of a human figure will be discussed as a first specificobject, and a natural plant area as a second specific object. The firstcolor determination unit 202 and the second color determination unit 204then obtain color information about the respective detected specificobjects. More specifically, the first color determination unit 202detects a first specific object from the captured image, and obtainscolor information about the specific object. The second colordetermination unit 204 detects a second specific object from thecaptured image, and obtains color information about the specific object.In the present exemplary embodiment, if a human skin area of a humanfigure is detected as the first specific object, the color informationabout the first specific object is skin color information about thehuman skin. If a natural plant is detected as the second specificobject, the color information about the second specific object is greeninformation about the plant. Details of the processing for obtaining thecolor information from the areas of the specific objects detected by thefirst color determination unit 202 and the second color determinationunit 204 will be described below.

The white determination unit 206 obtains color information about an areaincluding other than the areas of the specific objects (specific objectareas) from the captured image. More specifically, the whitedetermination unit 206 obtains color information about an area includingother than the two or more specific objects detected by the first colordetermination unit 202 and the second color determination unit 204. Aswill be described in detail below, in the present exemplary embodiment,the white determination unit 206 obtains white information from the areaincluding other than the specific object areas.

The first color estimation unit 203 estimates a first light source colorbased on the color information about the first specific object detectedby the first color determination unit 202. The estimated first lightsource color will hereinafter be referred to as a first estimated lightsource color.

The second color estimation unit 205 estimates a second light sourcecolor based on the color information about the second specific objectdetected by the second color determination unit 204. The estimatedsecond light source color will hereinafter be referred to as a secondestimated light source color.

The third color estimation unit 207 estimates a third light source colorbased on the color obtained by the white determination unit 206, i.e.,the white information about the area including other than specificobject areas of the captured image. The estimated third light sourcecolor will hereinafter be referred to as a third estimated light sourcecolor.

Details of the processing by the first, second, and third colorestimation units 203, 205, and 207 will be described below.

The degree of similarity calculation unit 208 calculates degrees ofsimilarity between the first, second, and third light source colors, andcalculates the degrees of reliability of the respective first, second,and third light source colors from the degrees of similarity.

The color mixing unit 209 performs color mixing processing for assigningweights to the respective first, second, and third light source colorsand mixing the weighted light source colors based on the degrees ofsimilarity or the degrees of reliability calculated from the degrees ofsimilarity.

The correction value calculation unit 210 calculates WB correctionvalues to be applied to the captured image based on a light source colorresulting from the color mixing processing by the color mixing unit 209.

In other words, in the present exemplary embodiment, the degrees ofsimilarity between the first to third light source colors arecalculated, and the degrees of reliability of the first to third lightsource colors are further calculated from the degrees of similarity. TheWB correction values are calculated by assigning weights to the first tothird light source colors based on the degrees of reliability. Detailsof the processing by the degree of similarity calculation unit 208, thecolor mixing unit 209, and the correction value calculation unit 210will be described below. The image processing apparatus 106 thenperforms the WB processing using the WB correction values.

FIG. 3 is a flowchart related to the processing by the CPU 103 of theimaging apparatus 100. In the present exemplary embodiment, the releasebutton is a two-level button. In the following description, SW1 of theoperation unit 110 refers to half-pressing of the release button, andSW2 of the operation unit 110 refers to full-pressing of the releasebutton.

In step S301, the CPU 103 accepts input information from the user viathe operation unit 110.

In step S302, the CPU 103 adjusts settings of the optical system 101,such as the focal length, shutter speed, and aperture value, based onthe input information from the user.

In step S303, the CPU 103 adjusts the photometric areas of thephotometric sensor 105 based on the input information from the user.

In step S304, the CPU 103 adjusts settings of the image sensor 102, suchas the ISO speed, based on the input information from the user.

In step S305, the CPU 103 displays information about the settingschanged by the processing of steps S302 to S304 on the display unit 109,and thereby presents the information to the user.

The order of steps S301 to S305 is not limited thereto, and may befreely changed depending on the processing.

In step S306, the CPU 103 determines whether there is SW1 input fromrelease button (whether SW1 is on or off). The CPU 103 repeats theoperation of steps S301 to S305 unless SW1 is input from the releasebutton (NO in step S306). On the other hand, if SW1 is input from therelease button (SW1 is on) (YES in step S306), the processing proceedsto step S307.

In step S307, the CPU 103 measures the brightness of the objects basedon the output from the photometric sensor 105. If the imaging apparatus100 is set to an automatic exposure (AE) mode, the CPU 103 automaticallyadjusts the exposure level based on the shutter speed, the aperturevalue, and the ISO speed.

In step S308, if the imaging apparatus 100 is set to an automatic focus(AF) mode, the CPU 103 adjusts focus by controlling a focus lens of theoptical system 101 based on information from a non-illustrated distancemeasurement sensor.

The order of steps S307 and S308 is not limited thereto, and may befreely changed depending on the processing.

In step S309, the CPU 103 determines whether there is SW2 input from therelease button (SW2 is on or off). The CPU 103 repeats the operation ofsteps S301 to S308 unless SW2 is input from the release button (NO instep S309). On the other hand, if SW2 is input from the release button(SW2 is on) (YES in step S309), the processing proceeds to step S310.

In step S310, the CPU 103 causes the image sensor 102 to performexposure, and stores raw data obtained by A/D-converting an imagingsignal of the image sensor 102 into the primary storage device 104.

In step S311, the image processing apparatus 106 detects specificobjects from the data stored in the primary storage device 104 under thecontrol of the CPU 103. As an example, the present exemplary embodimentdeals with a case where human skin is detected as a first specificobject and a natural green as a second specific object. The specificobjects to be detected are not limited thereto, and any specific objectsof which original color can be estimated can be used. Examples include aface, a blue sky, a cloud, ground, a tree trunk, asphalt pavement, anautumn leaf, and a dead leaf. While the present exemplary embodimentdeals with a case of detecting two specific objects, three or morespecific objects may be detected. The method for detecting the specificobjects may use rule-based techniques as well as deep learningtechniques.

In step S312, the image processing apparatus 106 calculates the WBcorrection values based on the data stored in the primary storage device104 under the control of the CPU 103. Details of the processing will bedescribed below.

In step S313, the image processing apparatus 106 performs developmentprocessing on the raw data stored in the primary storage device 104 byusing the WB correction values calculated in step S312, under thecontrol of the CPU 103.

In step S314, the CPU 103 displays an image based on the image dataobtained by the development processing, on the display unit 109.

In step S315, the CPU 103 records the image data obtained by thedevelopment processing in the recording medium 107.

Details of the calculation of the WB correction values by the imageprocessing apparatus 106 in step S312 will now be described.

FIG. 4 is a flowchart of the WB correction value calculation processingperformed by the functional units of the image processing apparatus 106illustrated in FIG. 2.

In step S401, the division unit 201 divides the image data stored in theprimary storage device 104 into given numbers of blocks horizontally andvertically as illustrated in FIG. 5, for example. The division unit 201calculates R, G. and B integral values in each block.

In step S402, the first color determination unit 202 detects a firstspecific object area by specific object detection processing, andcalculates only the R. G, and B integral values of the blocks in thefirst specific object area. In the present exemplary embodiment, a humanskin area is detected as the first specific object area, and the R, G,and B integral values in the human skin area are calculated. The firstcolor determination unit 202 further calculates an R/G value and a B/Gvalue from the R, G, and B integral values.

The first color estimation unit 203 then estimates the R/G and B/Gvalues of the light source color from the R/G and B/G valuescorresponding to the human skin area, calculated by the first colordetermination unit 202.

A method for estimating the R/G and B/G values of the light source colorfrom the R/G and B/G values corresponding to the human skin area willnow be described.

FIG. 6A is a graph of the R/G and B/G values. The horizontal axisindicates the R/G value, and the vertical axis the B/G value. The solidline in the graph represents a black-body radiation locus 601.

In the present exemplary embodiment, the secondary storage device 108stores a correspondence table of various types of light source colorsand the R/G and B/G values of human skin under the respective types oflight sources in advance. While the color of actual human skin variesfrom one person to another, the correspondence table is generated withskin color near the center of such variations as a representative value.

In FIG. 6A, suppose, for example, that the R/G and B/G valuescorresponding to the human skin area fall on the position of a circle602. In such a case, the light source color corresponding to the colorof the human skin area obtained from the correspondence table is locatedat the position of a circle 603. Similarly, if, for example, the R/G andB/G values corresponding to the human skin area fall on the position ofa circle 604, the light source color corresponding to the color of thehuman skin area obtained from the corresponding table is located at theposition of a circle 605. In such a manner, the first color estimationunit 203 estimates the light source color from the color of the humanskin area by using the R/G and B/G values of the human skin areacalculated by the first color determination unit 202 and thecorresponding table. In the present exemplary embodiment, the lightsource color estimated by the first color estimation unit 203 is thefirst estimated light source color.

In step S403, the second color determination unit 204 detects a secondspecific object area by specific object detection processing, andcalculates only the R, G, and B integral values of the blocks in thesecond specific object area. In the present exemplary embodiment, anatural green area is detected as the second specific object area, andthe R, G, and B integral values of the natural green area arecalculated. The second color determination unit 204 also calculates R/Gand B/G values from the R, G. and B integral values.

The second color estimation unit 205 then estimates the R/G and B/Gvalues of the light source color from the R/G and B/G valuescorresponding to the natural green area, calculated by the second colordetermination unit 204.

A method for estimating the RIG and B/G values of the light source colorfrom the R/G and B/G values corresponding to the natural green area willnow be described.

FIG. 6B is a graph of the R/G and B/G values. The horizontal axisindicates the R/G value, and the vertical axis the B/G value. The solidline in the graph represents the black-body radiation locus 601.

In the present exemplary embodiment, a corresponding table stored in thesecondary storage device 108 in advance contains various types of lightsource colors and the R/G and B/G values of a natural green under therespective types of light sources in association with each other. Whileactual natural greens vary in color, the corresponding table isgenerated with color near the center of such variations as arepresentative value.

In FIG. 6B, if, for example, the R/G and B/G values corresponding to thenatural green area falls on the position of a circle 606, the lightsource color corresponding to the color of the natural green areaobtained from the corresponding table is located at the position of acircle 607. Similarly, if, for example, the R/G and B/G valuescorresponding to the natural green area falls on the position of acircle 608, the light source color corresponding to the color of thenatural green area obtained from the corresponding table is located atthe position of a circle 609. In such a manner, the second colorestimation unit 205 estimates the light source color from the color ofthe natural green area by using the R/G and B/G values of the naturalgreen area calculated by the second color determination unit 204 and thecorresponding table. In the present exemplary embodiment, the lightsource color estimated by the second color estimation unit 205 is thesecond estimated light source color.

In step S404, the white determination unit 206 detects a white area fromthe entire captured image, and calculates the R. G, and B integralvalues in the white areas. The white determination unit 206 furthercalculates the R/G and B/G values of the white area from the R, G, and Bintegral values.

FIG. 6C is a graph of the R/G and B/G values. The horizontal axisindicates the R/G value, and the vertical axis the B/G value. The areasurrounded by the dotted line in the graph represents a white area 610.The white determination unit 206 integrates the R, G. and B values ofthe blocks included in the white area 610 set in the R/G-B/G coordinatesystem illustrated in FIG. 6C, and calculates a set of R, G, and Bintegral values represented by a circle 611. The white area 610illustrated in FIG. 6C is set to include the vicinity of the black-bodyradiation locus 601 and the light source colors of a fluorescent lampand a mercury lamp.

The third color estimation unit 207 estimates the R/G and B/G values ofthe light source color from the R/G and B/G values corresponding to thewhite area 610 set by the white determination unit 206. The R/G and B/Gvalues estimated by the third color estimation unit 207 indicate thelight source color calculated from the entire captured image (i.e.,areas including the specific object areas). In the present exemplaryembodiment, the light source color estimated by the third colorestimation unit 207 is the third estimated light source color.

While in the present exemplary embodiment, the white determination unit206 applies the processing to the entire captured image, the processingof the white determination unit 206 may be applied to only the areasexcluding the first and second specific object areas.

In the present exemplary embodiment, the method for estimating the lightsource color by extracting the blocks included in the white area 610from the image has been described. However, other techniques for lightsource color estimation processing may be used as long as data on animage area including other than the first and second specific objectareas is used.

In step S405, the degree of similarity calculation unit 208 calculatesthe degrees of similarity between the first to third estimated lightsource colors, and further calculates the degrees of reliability of thefirst to third estimated light source colors from the calculated degreesof similarity. Details will be described with reference to FIGS. 7A to7C.

In FIGS. 7A to 7C, the solid line represents the black-body radiationlocus 601. A circle 701 indicates the position of the R/G and B/G valuesof the foregoing first estimated light source color, a circle 702 thatof the second estimated light source color, and a circle 703 that of thethird estimated light source color. Here, the degree of similaritycalculation unit 208 calculates the distances between the respectivepositions of the R/G and B/G values of the first to third estimatedlight source colors on the graphs.

For example, the distances can be calculated by the following Eqs. (1):

D12=((R/G1−R/G2){circumflex over ( )}2+(B/G1−B/G2){circumflex over( )}2){circumflex over ( )}(½),

D13=((R/G1−R/G3){circumflex over ( )}2+(B/G1−B/G3){circumflex over( )}2){circumflex over ( )}(½), and

D23=((R/G2−R/G3){circumflex over ( )}2+(B/G2−B/G3){circumflex over( )}(½)  Eqs. (1)

In Eqs. (1), D12 is the distance between the positions of the R/G andB/G values of the first and second estimated light source colors on thegraph. Similarly, D13 is the distance between those of the first andthird estimated light source colors, and D23 the distance between thoseof the second and third estimated light source colors. R/G1 and B/G1 inEqs. (1) are the R/G and B/G values of the first estimated light sourcecolor. Similarly, R/G2 and B/G2 are the R/G and B/G values of the secondestimated light source color, and R/G3 and B/G3 the R/G and B/G valuesof the third estimated light source color.

The degree of similarity calculation unit 208 determines that thedegrees of similarity are high if the distances fall within apredetermined value. On the other hand, the degree of similaritycalculation unit 208 determines that the degrees of similarity are lowif the distances exceed the predetermined value. For example, if thedistance D12 falls within the predetermined value, the degree ofsimilarity calculation unit 208 determines that the degree of similaritybetween the first and second estimated light source colors is high. Forexample, if the distance D12 exceeds the predetermined value, the degreeof similarity calculation unit 208 determines that the degree ofsimilarity between the first and second estimated light source colors islow. The degree of similarity calculation unit 208 also similarlydetermines the degrees of similarity using the distance D13 and thedistance D23. The degree of similarity calculation unit 208 thendetermines the degrees of reliability of the first to third estimatedlight source colors based on the degrees of similarity.

FIG. 7A illustrates a case where all the degrees of similarity betweenthe first, second, and third estimated light source colors are high. Insuch a case, the degree of similarity calculation unit 208 determinesthat all the estimated light source colors have a high degree ofreliability, since all the degrees of similarity between the estimatedlight source colors are high.

FIG. 7B illustrates an example where the degree of similarity betweenthe first and third estimated light source colors is high while thedegrees of similarity of the second estimated light source color withboth the first and third estimated light source colors are low. If onlyone of the estimated light source colors is thus at large distances fromthe other two, the degree of similarity calculation unit 208 determinesthat the detect result of the specific object is likely to be erroneousand the one estimated light source color with low degrees of similarity(in this example, the second estimated light source color) has a lowdegree of reliability.

FIG. 7C illustrates an example where all the degrees of similaritybetween the first, second, and third estimated light source colors arelow. If all the three estimated light source colors are thus at largedistances, the degree of similarity calculation unit 208 determines thatthe degrees of reliability are neutral, since which estimated lightsource color has a high degree of reliability based on the estimationresult is unable to be determined. The degree of similarity calculationunit 208 also determines the degrees of reliability of the first,second, and third estimated light source colors to be neutral if atleast either of the first and second specific objects is not detected.

FIG. 8 is a chart illustrating a correspondence table for determiningthe degrees of reliability from a combination of the degrees ofsimilarity between the first, second, and third estimated light sourcecolors as described above. The degree of similarity calculation unit 208determines the degrees of reliability of the estimated light sourcecolors based on the correspondence table illustrated in FIG. 8. In theexample of FIG. 7A, all the degrees of similarity of the first to thirdestimated light source colors in FIG. 8 are “high”, and the degree ofsimilarity calculation unit 208 determines that all the degrees ofreliability of the first to third estimated light source colors are“high”. In the example of FIG. 7B, the degrees of similarity of thefirst and third estimated light source colors are “high” and that of thesecond estimated light source color is “low” in FIG. 8. In such a case,the degree of similarity calculation unit 208 determines that thedegrees of reliability of the first and third estimated light sourcecolors are “high” and the degree of reliability of the second estimatedlight source color is “low”. In the example of FIG. 7C, all the degreesof similarity of the first to third estimated light source colors inFIG. 8 are “low”, and the degree of similarity calculation unit 208determines that the degrees of reliability of the first to thirdestimated light source colors are “neutral”.

In step S406, the degree of similarity calculation unit 208 determineswhether the present processing is intended to capture the first frameafter power-on of the imaging apparatus 100.

If the present processing is intended to capture the first frame (YES instep S406), the processing proceeds to step S407. In step S407, thedegree of similarity calculation unit 208 stores the degrees ofreliability of the first to third estimated light source colorscalculated by the processing so far and the R/G and B/G values of theestimated light source colors into the primary storage device 104.

In step S408, the color mixing unit 209 performs color mixing processingbased on the degrees of reliability of the first to third estimatedlight source colors. Details of the color mixing processing by the colormixing unit 209 will now be described.

The degrees of reliability of the estimated light source colors areexpressed in terms of “high”, “neutral”, and “low” as illustrated in thecorrespondence table of FIG. 8.

If any of the first to third estimated light source colors has a degreeof reliability “high”, the color mixing unit 209 mixes the estimatedlight source colors by Eqs. (2) to be described below, with the weightassigned to the estimated light source color(s) having the degree ofreliability “high” as 100%, and the weight assigned to the estimatedlight source color(s) having the other degrees of reliability (“neutral”and “low”) as 0%.

If none of the first to third estimated light source colors has a degreeof reliability “high” or “neutral” but all the first to third estimatedlight source colors have a degree of reliability “low”, the color mixingunit 209 uses a mixed light source color obtained in capturing theprevious frame. Note that the imaging apparatus 100 according to thepresent exemplary embodiment is controlled so that the degrees ofreliability of the first to third estimated light source colors are notset to “low” in capturing the first frame after power-on.

In the present exemplary embodiment, determination errors in theestimated light source colors can thus be determined by comparing therelationship between the distances between the respective estimatedlight source colors.

R/G0=(R/G1×a+R/G2×b+R/G3×c)/(a+b+c), and

B/G0=(B/G1×a+B/G2×b+B/G3×c)/(a+b+c)  Eqs. (2)

R/G0 in Eqs. (2) is the R/G value after the color mixing processing, andB/G0 the B/G value after the color mixing processing. R/G1 and B/G1 inEqs. (2) are the R/G and B/G values of the first estimated light sourcecolor, R/G2 and B/G2 the R/G and B/G values of the second estimatedlight source color, and R/G3 and B/G3 the R/G and B/G values of thethird estimated light source color. In Eqs. (2), a is the weight to theR/G and B/G values of the first estimated light source color, b theweight to the R/G and B/G values of the second estimated light sourcecolor, and c the weight to the R/G and B/G values of the third estimatedlight source color.

The color mixing unit 209 stores R/G0 and B/G0 of the light source colorcalculated here into the primary storage device 104 for use insubsequent frames. While in the present exemplary embodiment the colormixing unit 209 performs the color mixing processing based on thedegrees of reliability, the color mixing unit 209 may perform colormixing processing based on the degrees of similarity.

In step S409, the correction value calculation unit 210 calculates theWB correction values from the R/G and B/G values (R/G0 and B/G0) of thelight source color after the color mixing processing calculated in stepS408 by using the following Eqs. (3):

Wr=1/(R/G0),

Wg=1,and

Wb=I/(B/G0)  Eqs. (3)

In Eqs. (3), Wr is the WB correction value for the R value, Wg the WBcorrection value for the G value, and Wb the WB correction value for theB value. R/G0 is the R/G value after the color mixing processing, andB/G0 the B/G value after the color mixing processing.

The calculation processing of the WB correction values by the correctionvalue calculation unit 210 ends.

Now, processing for the case where the present processing is determinedto be intended to capture the second or a subsequent frame after thepower-on of the imaging apparatus 100 in step S406 will be described.

If the processing is determined to be intended to capture the second ora subsequent frame after the power-on of the imaging apparatus 100 instep S406 (NO in step S406), the processing proceeds to step S410.

In step S410, the degree of similarity calculation unit 208 recalculatesthe degrees of reliability of the first to third estimated light sourcecolors based on comparison results of the degrees of reliability of thefirst to third estimated light source colors between the latest frameand a past frame.

Specifically. the degree of similarity calculation unit 208 compares theR/G and B/G values of the first to third estimated light source colorsdetermined in the latest frame with those in a past frame. Ifdifferences of the foregoing distances between the positions of the R/Gand B/G values of the first to third estimated light source colorsobtained in the latest frame from those in the past frame fall within apredetermined value, the degree of similarity calculation unit 208recalculates the degrees of reliability and the estimated light sourcecolors as in the table illustrated in FIG. 9. If the distances betweenthe positions of the R/G and B/G values of the estimated light sourcecolors in the latest frame differ from those in the past frame by morethan the predetermined value, the degree of similarity calculation unit208 uses the degrees of reliability and the estimated light sourcecolors determined in the latest frame as the recalculated degrees ofsimilarity and the recalculated estimated light source colors.

Details of the table illustrated in FIG. 9 will now be described.

As a basic operation in FIG. 9, the degree of similarity calculationunit 208 recalculates the degrees of reliability and the estimated lightsource colors with higher priority to the degrees of reliabilitycalculated from the latest frame. If a degree of reliability calculatedfrom the latest frame is “neutral”, the degree of similarity calculationunit 208 recalculates the degree of reliability and the estimated lightsource color with higher priority given to the degree of reliabilitycalculated from the past frame. If both the degrees of reliabilitycalculated in the past frame and the latest frame are “neutral”, thedegree of similarity calculation unit 208 uses the degree of reliabilityand the estimated light source color in the latest frame as therecalculation result.

As described above, in the present exemplary embodiment, the degrees ofreliability and the estimated light source colors are recalculated basedon the comparison results of the distances (degrees of similarity)between the positions of the R/G and B/G values and the degrees ofreliability of the estimated light source colors calculated in thelatest frame with those calculated in the past frame. More specifically.in the present exemplary embodiment, at least one of the degrees ofsimilarity and the degrees of reliability of the first, second, andthird estimated light source colors is compared with the same type ofresult obtained in capturing the next or a subsequent frame. The WBcorrection values for the image of the next or subsequent frame arecalculated based on the comparison result. Whether the estimated lightsource colors are erroneous can thus be determined even if the degreesof reliability calculated in the latest frame are determined to be“neutral”.

Now, suppose that when the degree of similarity calculation unit 208determines whether the present processing is intended to capture thesecond or a subsequent frame after the power-on of the imaging apparatus100 in step S406, a predetermined time has elapsed or the brightness ofan object calculated by the photometric sensor 105 has changed by apredetermined value or more since the previous frame. In such a case,the reliability of past frame information drops. The degree ofsimilarity calculation unit 208 may therefore be configured to processthe current frame as the first frame without using information obtainedfrom the previous frame in at least either one of the cases where thepredetermined time has elapsed and where the brightness has changed bythe predetermined value or more.

As described above. according to the present exemplary embodiment, theeffect of erroneous WB corrections due to misdetection of objects can bereduced by automatic WB calculation using specific object detection.This enables appropriate WB control and can prevent automatic WBaccuracy from dropping.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure includes exemplary embodiments, it is to beunderstood that the disclosure is not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2020-086873, filed May 18, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising one ormore processors configured to function as following units: a detectionunit configured to detect two or more specific objects from a capturedimage; a first color estimation unit configured to estimate a firstlight source color from color information about a first specific objectamong the detected specific objects; a second color estimation unitconfigured to estimate a second light source color from colorinformation about a second specific object among the detected specificobjects; a third color estimation unit configured to estimate a thirdlight source color from color information about an area including otherthan the detected specific objects; and a calculation unit configured tocalculate a white balance correction value to be applied to the capturedimage by assigning a weight to at least one of the first, second, andthird light source colors based on degrees of similarity between thefirst, second, and third light source colors.
 2. The image processingapparatus according to claim 1, wherein the calculation unit isconfigured to calculate degrees of reliability of the respective first,second, and third light source colors based on the degrees of similarityof the first, second, and third light source colors, and calculate thewhite balance correction value by assigning the weight(s) based on thedegrees of reliability.
 3. The image processing apparatus according toclaim 2, wherein the calculation unit is configured to: determinewhether the degrees of similarity of the first, second, and third lightsource colors are high or low by comparing the degrees of similaritywith a predetermined value, in a case where the degrees of similarity ofall the first, second, and third light source colors are determined tobe high, determine that the degrees of reliability of the first, second,and third light source colors are high, in a case where the degree ofsimilarity of only one of the first, second, and third light sourcecolors is determined to be low, determine that the degree of reliabilityof the light source color corresponding to the degree of similaritydetermined to be low is low, and in a case where the degrees ofsimilarity of all the first, second, and third light source colors aredetermined to be low, determine that the degrees of reliability of thefirst, second, and third light source colors are neutral.
 4. The imageprocessing apparatus according to claim 3, wherein the calculation unitis configured to calculate the white balance correction value based on alight source color obtained by performing mixing processing on thefirst, second, and third light source colors after the weight(s) is/areassigned.
 5. The image processing apparatus according to claim 4,wherein the calculation unit is configured to perform the mixingprocessing with the weight(s) to a light source color or colors of whichthe degree(s) of reliability is(are) determined to be high among thefirst, second, and third light source colors as 100% and the weight(s)to a light source color or colors of which the degree(s) of reliabilityis(are) not determined to be high as 0%.
 6. The image processingapparatus according to claim 4, wherein the calculation unit isconfigured to, in a case where the degrees of reliability of none of thefirst, second, and third light source colors are determined to be highand the degrees of reliability of all the light source colors aredetermined to be neutral or low, perform the mixing processing with theweight(s) to a light source color or colors of which the degree(s) ofreliability is(are) determined to be neutral as 100% and the weight(s)to a light source color or colors of which the degree(s) of reliabilityis(are) not determined to be neutral as 0%.
 7. The image processingapparatus according to claim 4, wherein the calculation unit isconfigured to, in a case where the degrees of reliability of none of thefirst, second, and third light source colors are determined to be highor neutral and the degrees of reliability of all the first, second, andthird light source colors are determined to be low, use informationabout the white balance correction value calculated based on a lightsource color obtained by the mixing processing on a captured image of aprevious frame.
 8. The image processing apparatus according to claim 7,wherein the calculation unit is configured to, in at least either one ofcases where a predetermined time has elapsed and where brightness of thespecific object has changed by a predetermined value or more sincecapturing of the image of the previous frame, not use the informationobtained in the previous frame.
 9. The image processing apparatusaccording to claim 2, wherein the calculation unit is configured to makea comparison between information about at least one of the first,second, and third light source colors, the degrees of similarity of thefirst, second, and third light source colors, and the degrees ofreliability of the first, second, and third light source colors and asame type of information obtained by capturing an image of a next orsubsequent frame, and calculate the white balance correction value forthe image of the next or subsequent frame based on a result of thecomparison.
 10. The image processing apparatus according to claim 1,wherein the detection unit is configured to detect at least two or moreof objects including a face, skin, plant green, a blue sky, a cloud,ground, a tree trunk, asphalt pavement, an autumn leaf, and a dead leafas the specific objects from the captured image.
 11. The imageprocessing apparatus according to claim 1, wherein the third colorestimation unit is configured to estimate the third light source colorfrom color information about a white area included in the area includingother than the specific objects.
 12. An image processing methodcomprising: detecting two or more specific objects from a capturedimage; estimating a first light source color from color informationabout a first specific object among the detected specific objects;estimating a second light source color from color information about asecond specific object among the detected specific objects; estimating athird light source color from color information about an area includingother than the detected specific objects; and calculating a whitebalance correction value to be applied to the captured image byassigning a weight to at least one of the first, second, and third lightsource colors based on degrees of similarity between the first, second,and third light source colors.
 13. A non-transitory computer-readablestorage medium storing a program for causing a computer to executefollowing steps comprising: detecting two or more specific objects froma captured image; estimating a first light source color from colorinformation about a first specific object among the detected specificobjects; estimating a second light source color from color informationabout a second specific object among the detected specific objects;estimating a third light source color from color information about anarea including other than the detected specific objects; and calculatinga white balance correction value to be applied to the captured image byassigning a weight to at least one of the first, second, and third lightsource colors based on degrees of similarity between the first, second,and third light source colors.